Using Cross-validation Methods
2024-07-28
CV widely used in various fields including:
Generalizability:
How well predictive models created from a sample fit other samples from the same population.
Overfitting:
When a model fits the the underlying patterns of the training data too well.
Model fits characteristics specific to the training set:
Hyperparameters:
Are model configuration variables
Subsets the data into K approximately equally sized folds
Split The Subsets into test and training sets
Repeat K Times
Calculate the mean error
| Method | Computation | Bias | Variance |
|---|---|---|---|
| K-Fold | Lower | Intermediate | Lower |
| LOOCV | Highest | Unbiased | High |
K-fold where K = 5 or K = 10 is recommended:
By measuring the quality of fit we can select the model that Generalizes best.
\[ \text{MAE} = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{f}(x_i)| \tag{1} \]
\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{f}(x_i))^2} \tag{2} \]
\[ \text{R}^2 = \frac{SS_{tot}-SS_{res}}{SS_{tot}} = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum_{i=1}^{n}(y_i - \hat{f}(x_i))^2}{\sum_{i=1}^{n}(y_i-\bar{f}(x_i))^2} \tag{3} \]
(James et al. 2013), (Hawkins, Basak, and Mills 2003), (Helsel and Hirsch 1993)
\[
CV_{(k)} = \frac{1}{k}\sum_{i=1}^{k} \text{Measuer of Errori}_i \tag{4}
\]
\[ CV_{(n)} = \frac{1}{n}\sum_{i=1}^{n} \text{Measuer of Errori}_i \tag{5} \]
(I-C Yeh 1998) modeled compression strength of high performance concrete (HPC) at various ages and made with different ratios of components. The data used for their study was made publicly available and can be downloaded UCI Machine Learning Repository (I-Cheng Yeh 2007).
All variables are quantitative
|
\[ \hat{Strength} = 28.258_\text{Cement + } 0.067_\text{Superplasticizer + } 0.872_\text{Age } 0.111_\text{Water} \]
| Measure of Error | Result |
|---|---|
| RMSE | 12.13 |
| MAE | 9.23 |
| R2 | 0.46 |
| Measure of Error | Result |
|---|---|
| RMSE | 12.13 |
| MAE | 9.23 |
| R2 | 0.46 |
| Measure of Error | Result |
|---|---|
| RMSE | 11.87 |
| MAE | 9.43 |
| R2 | 0.49 |
| Measure of Error | Result |
|---|---|
| RMSE | 8.73 |
| MAE | 6.82 |
| R2 | 0.73 |
| Measure of Error | Result |
|---|---|
| RMSE | 8.73 |
| MAE | 6.82 |
| R2 | 0.73 |
| Measure of Error | Result |
|---|---|
| RMSE | 5.93 |
| MAE | 4.32 |
| R2 | 0.87 |
| Measure of Error | Result |
|---|---|
| RMSE | 8.27 |
| MAE | 6.39 |
| R2 | 0.75 |
| Method | Measure of Error | Linear Regression | LightGBM |
|---|---|---|---|
| 5-Fold | RMSE | 12.13 | 8.73 |
| 5-Fold | MAE | 9.23 | 6.82 |
| 5-Fold | R2 | 0.46 | 0.73 |
| LOOCV | RMSE | 12.13 | 5.93 |
| LOOCV | MAE | 9.23 | 4.32 |
| LOOCV | R2 | 0.46 | 0.87 |
| NCV | RMSE | 11.87 | 8.27 |
| NCV | MAE | 9.43 | 6.39 |
| NCV | R2 | 0.49 | 0.75 |
All figures were created by the authors.